Systems and methods to estimate human length

ABSTRACT

Systems and methods use a three-dimensional snapshot of an infant, child, adolescent, or adult where the subject may have bent legs and an unaligned head position, to provide an estimate of human length. The system identifies anatomical features from the digital information and generates a virtual skeleton. The cumulative distances between pseudo-joints of the virtual skeleton provide an estimate of human length. Comparing length estimates from multiple three-dimensional snapshots of the same individual acquired over time provide an indication of growth of the infant, child, or adolescent. Daily estimates of length can detect growth faltering sooner than less frequent estimates of length, which can lead to timely intervention.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

BACKGROUND

Indications of growth in infants, children, and adolescents are used toevaluate physical development and are important gauges of overallhealth. Length is a key indicator of growth. For measuring length of aninfant, an infantometer is used by trained personnel to measure thecrown-to-heel length. Use of the infantometer involves placing theinfant on its back on a plate, straightening the legs such that theinfant's foot rests on a foot plate, aligning the head to rest on a headplate, and measuring the length of the infant with a ruler attached tothe infantometer. This process causes distress in the infant. Reluctanceby medical personnel to completely stretch the infant could lead toinaccurate measurements, and variation within and across personnel couldlead to unreliable measurements. For measuring children, adolescents,and adults, a stadiometer is used. Use of the stadiometer involveshaving the child, adolescent, or adult stand on a plate or the floor,with the back to a perpendicular wall or rod, assuming an uprightstance, and looking forward with the head in the Frankfurt plane. Theprocess to ensure upright posture and the correct head position requirestrained personnel to measure crown-to-heel length, and variation withinand across personnel could lead to unreliable measurements.

SUMMARY

A reliable measure of the growth as the change in length of a developinginfant, child, or adolescent can be estimated from computer processingof data from optical sensors that provide information sufficient tocreate three-dimensional images. One way to accomplish this is withsophisticated algorithms to transform these images into repeatableestimates of a pseudo skeleton. This not an instance of bone structurebut a framework that connects a set of simulations of joints such asankles, knees, hips and neck that are located in three dimensional spacehereafter referred to as 3D. These features, hereafter referred to asanatomical landmarks, may be combined such that a summation of distancesalong the length of the resulting stick figure may provide an accurateestimate of length of an infant, child, adolescent, or adult. Anotherway to accomplish this is by aligning the head position andstraightening the legs so a point or location on the head and a point orlocation on the legs are on a straight line that provides an accurateestimate of length on a subject, where the subject can be, for example,an infant, child, adolescent or adult.

Systems and methods are disclosed to provide an indication of growth bymeasuring changes in length between anatomical landmarks located by oneor more computational algorithms on a 3D image. These partial lengthestimates are summed to form an overall length estimate that isindependent of small postural variation.

Further, system and methods are disclosed to acquire a three-dimensionalimage of a subject in a natural supine or upright position from asnapshot, such as a digital snapshot can be taken of the frontal surfaceof the subject using a time-of-flight camera. In an embodiment, theacquired 3D image is noisy as there are limitations to the accuracy andresolution of any measurement. However a computational algorithm thatemphasizes detection and use of large anatomical features that arepoints of inflection where the body naturally bends can be used tominimize the noise. The systems and methods can use digital filteringand point-cloud processing to identify anatomical features of an infant,child, adolescent, or adult from the acquired 3D image. A point-cloud isa set of points in 3D space. Point-cloud processing can be characterizedas computational manipulation of the point-cloud of an object togenerate a 3D surface representing the object. The identified anatomicalfeatures can be used to provide estimations of total crown-to-heellength or partial length between natural landmarks on the body, wherepartial length is the distance between at least two natural landmarksand is less than the total length. Comparisons over a period of time ofthe estimated lengths can provide an indication of growth, which dependson the stage of development when an individual is growing (i.e., ininfancy, childhood, or adolescence). The measures of length can providean indication of whether the infant, child, or adolescent is developingor growing normally. Advantageously, point-cloud processing can identifythe anatomical features for accurate and reliable estimation of thelength from the noisy 3D images, despite bent legs, unaligned headpositions, and different postures in images acquired at different times.

Aspects of the disclosure relate to a computer-implemented method toestimate length of a subject from a three-dimensional image. Thecomputer-implemented method comprises acquiring a three-dimensionalimage of a subject; processing the three-dimensional image of thesubject to generate a digital representation of the subject; andprocessing the digital representation of the subject to determine atleast a partial length of the subject, wherein the subject is an infant,child, adolescent, or adult.

In an embodiment, processing the digital representation of the subjectto determine at least the partial length of the subject comprisesprocessing the digital representation of the subject to generate avirtual skeleton of the subject; and measuring distance along at least aportion of the virtual skeleton to provide an estimate of at leastpartial length of the subject.

In an embodiment, processing the digital representation of the subjectto generate the virtual skeleton of the subject comprises locating aplurality of peaks and troughs in a point cloud representation of thesubject; locating a natural landmark associated with human anatomy ineach of ones of at least a portion of the plurality of peaks andtroughs; fitting spheres to model the located natural landmarks; andlocating centers of spheres to form the virtual skeleton.

In an embodiment, measuring distance along at least the portion of thevirtual skeleton comprises measuring distance between at least twosphere centers along the virtual skeleton to provide the estimate of atleast partial length of the subject. In another embodiment, locating theplurality of peaks and troughs comprises determining derivatives inlength and width planes of the point cloud representation of thesubject. In a further embodiment, fitting the spheres comprisesdetecting a set of extrema on surface of the point cloud representationof the subject and sizing the spheres based on the detected set ofextrema.

In an embodiment, the digital representation of the subject comprises apoint cloud representation of the subject. In another embodiment, thethree-dimensional image is acquired by a time of flight device. In afurther embodiment, the three-dimensional image is generated from aplurality of two-dimensional images.

In an embodiment, the three-dimensional image of the subject comprises athree-dimensional image of the subject on a mat that has defined edgesand the digital representation of the subject comprises a digitalrepresentation of the subject without the mat.

Aspects of the disclosure relate to a system to estimate length of asubject from a three-dimensional image. The system comprises one or morecomputing devices associated with a measurement system, where thesubject measurement system is configured to acquire a three-dimensionalimage of a subject; process the three-dimensional image of the subjectto generate a digital representation of the subject; and process thedigital representation of the subject to determine at least a partiallength of the subject, wherein the subject is an infant, child,adolescent, or adult.

Aspects of the disclosure relate to a computer-implemented method toestimate length of a subject from a three-dimensional image, thecomputer-implemented method comprising acquiring, from a time-of-flightdevice, a three-dimensional image of a subject on a mat that has definededges; processing the three-dimensional image to generate a point cloudrepresentation of the subject without the mat; locating a plurality ofpeaks and troughs in the point cloud representation of the subject;locating a natural landmark associated with human anatomy in each ofones of at least a portion of the plurality of peaks and troughs;fitting spheres to model the located natural landmarks; locating centersof spheres to form a virtual skeleton; and measuring distance between atleast two sphere centers along the virtual skeleton to provide anestimate of at least partial subject length.

In an embodiment, the mat has known dimensions. In another embodiment,the defined edges comprise one or more defined corners. In a furtherembodiment, the time-of-flight device comprises a time-of-flight camera.

In an embodiment, the three-dimensional image includes a depth image andan amplitude image of brightness. In another embodiment, thethree-dimensional digital image includes an amplitude value associatedwith the brightness information and a distance value associated with thedepth information. In a further embodiment, processing thethree-dimensional image to generate a point cloud representation of thesubject without the mat comprises smoothing the depth image with one ormore spatial filters to generate one or more secondary images; rotatinga view perspective of each of the one or more secondary images to aligna plane of the time-of-flight device with the plane of the mat; removingpoints on the one or more rotated secondary images with values less thanan estimated noise threshold to remove the mat; and combining the one ormore mat-removed secondary images to form the point cloud representationof the subject.

In an embodiment, locating the peaks and troughs comprises determiningderivatives in the length and width planes of the point cloudrepresentation to located peaks and troughs. In another embodiment,fitting the spheres comprises detecting a set of extrema on surface ofpoint cloud representation of the subject and sizing the spheres basedon the detected set of extrema.

In an embodiment, the computer-implemented method further comprisesdetermining an indication of growth by comparing a first estimate ofgrowth from a first three-dimensional image taken at a first earliertime with a second estimate of growth from a second three-dimensionalimage taken at a second later point in time.

Aspects of the disclosure relate to a system to estimate length of asubject from a three-dimensional image. The system comprises one or morecomputing devices associated with a measurement system, wherein themeasurement system is configured to acquire, from a time-of-flightdevice, a three-dimensional image of a subject on a mat that has definededges; process the three-dimensional image to generate a point cloudrepresentation of the subject without the mat; locate a plurality ofpeaks and troughs in the point cloud representation of the subject;locate a natural landmark associated with human anatomy in each of onesof at least a portion of the plurality of peaks and troughs; fit spheresto model the located natural landmarks; locate centers of spheres toform a virtual skeleton; and measure distance between at least twosphere centers along the virtual skeleton to provide an estimate of atleast partial length.

In an embodiment, processing the three-dimensional image to generate apoint cloud representation of the subject without the mat comprisessmoothing the depth image with one or more spatial filters to generateone or more secondary images. In another embodiment, processing thethree-dimensional image to generate a point cloud representation of thesubject without the mat further comprises rotating a view perspective ofeach of the one or more secondary images to align a plane of thetime-of-flight device with the plane of the mat. In a furtherembodiment, processing the three-dimensional image to generate a pointcloud representation of the subject without the mat further comprisesremoving points on the one or more rotated secondary images with valuesless than an estimated noise threshold to remove the mat. In a yetfurther embodiment, processing the three-dimensional image to generate apoint cloud representation of the subject without the mat furthercomprises combining the one or more mat-removed secondary images to formthe point cloud representation of the subject.

In an embodiment, locating the peaks and troughs comprises determiningderivatives in the length and width planes of the point cloudrepresentation to located peaks and troughs. In another embodiment,fitting the spheres comprises detecting a set of extrema on surface ofpoint cloud representation of the subject and sizing the spheres basedon the detected set of extrema. In a further embodiment, the measurementsystem is further configured to determine an indication of growth bycomparing a first estimate of growth from a first three-dimensionalimage taken at a first earlier time with a second estimate of growthfrom a second three-dimensional image taken at a second later point intime.

For purposes of summarizing the disclosure, certain aspects, advantagesand novel features of the inventions have been described herein. It isto be understood that not necessarily all such advantages may beachieved in accordance with any particular embodiment of the invention.Thus, the invention may be embodied or carried out in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other advantages as may be taughtor suggested herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, reference numbers are re-used to indicatecorrespondence between referenced elements. The drawings, associateddescriptions, and specific implementation are provided to illustrateembodiments for measuring the length of an infant, but does not to limitthe scope of the disclosure to other stages of development (i.e., tomeasures of a child, adolescent, or adult). A subject can be an infant,child, adolescent, or adult.

FIG. 1A illustrates a block diagram of an example system to acquire andprocess 3D digital images of a subject to provide an estimate of atleast partial length of the subject using networked computer system,according to certain embodiments.

FIG. 1B illustrates a block diagram of another example system to acquireand process 3D digital images of subject to provide an estimate of atleast partial length of the subject in a cell phone implementation,according to certain embodiments.

FIG. 2 illustrates a routine to form a point-cloud representation ofinfant subject using two digital images and fixed camera position,according to certain embodiments.

FIG. 2A is an example pictorial representation acquired by the imageprocessing and length estimation system from the image acquisitiondevice of the subject on the background, according to certainembodiments.

FIG. 2B an example pictorial representation of an example transformedimage of the subject represented by the distance of each point of thesurface of the subject to the floor, according to certain embodiments.

FIG. 3 illustrates a routine to form a point-cloud representation of aninfant using a single 3D digital image of the subject, according tocertain embodiments.

FIG. 3A is an example pictorial representation of a depth image of thesubject on the background, according to certain embodiments.

FIG. 3B is an example pictorial representation of the depth image ofFIG. 3A rotated to horizontal, according to certain embodiments.

FIG. 3C is an example pictorial representation of an amplitude image ofthe subject on the background, according to certain embodiments

FIGS. 3D and 3E are example pictorial representations of transformedimages of the subject representing the surface of the subject's body,according to certain embodiments.

FIG. 3F is an example pictorial representation of a surface meshrepresentation of the subject, according to certain embodiments.

FIG. 4 illustrates a routine to generate a pseudo-skeleton of an infant,according to certain embodiments.

FIG. 4A is an example pictorial representation of a point-cloudrepresentation showing the Y-axis, the X-axis, and Z=0, according tocertain embodiments.

FIGS. 4B-4G illustrate example pseudo-skeletons with sphericalrepresentations of natural landmarks, according to certain embodiments.

FIG. 5 illustrates a routine to provide an indication of growth,according to certain embodiments.

FIG. 6 illustrates an example time-of-flight system to capture a single3D image of infant subject used to generate a pseudo-skeleton of thesubject, according to certain embodiments.

FIGS. 7A and 7B illustrate front and back views of an example of acommercially available TOF camera system used to capture the 3D image ofa subject, according to certain embodiments.

FIG. 8 illustrates front and back views of an example of a stereo cameraand structured light system used to capture 3D images of the subject,according to certain embodiments.

DETAILED DESCRIPTION

Auxology can be described as the science of human growth anddevelopment. Traditionally, human growth has been characterized as acontinuous process because traditional auxology studies measure heightand weight with relatively long periods of time between consecutivemeasurements, such as every few month for infants and biannually orannually for children and adolescents, for example. When the growthmeasurement data, such as the height and weight, are acquired a fewtimes a year, each data point of the growth measurement data can becharacterized as the summation of many individual growth periods.Summing many individual growth periods in each observation and using thedistance between them as an increase in length may be misinterpreted ascontinuous growth between the observations.

Other human growth models propose that growth can be sporadicmini-growth spurts occurring approximately every 10-15 days, calledsaltations, each followed by a period of non-growth, called stasis. Thecharacteristics of individual saltations and stasis periods are lostwhen growth is measured infrequently. To determine the frequency andamplitude of saltations, it is desirable to measure infant length daily.The frequency and amplitude of the saltations determined from dailymeasurements of infant length can be used to better understand thebiology of human growth problems, such as deficits in amplitude ofsaltations, deficits in the frequency of saltations, and long periods ofstasis. Further, daily measurements of infant length can detect growthfaltering sooner than infrequent measurements, which can lead to timelyintervention.

Traditional methods to measure infant length can be invasive,cumbersome, and expensive. For example, one method to measure the lengthof an infant is for trained technicians to place the infant on aninfantometer, which is a device with a flat base and perpendicularheadboard and footboard than can be adjusted to show the distancebetween them in millimeters. Typically, two technicians are required toobtain accurate and reliable measures of length.

The infantometer protocol is complex and difficult to follow. Typicallythe two technicians (a) place the infant on its back on the flat base,(b) align the infant's head to touch the headboard with the eyes in the“Frankfurt plane”, (c) straighten the infant's legs horizontally andposition the feet vertically, (d) adjust the footboard until it touchesthe infant's feet and heels, and (e) read and record the distancebetween the headboard and footboard. The Frankfurt plane is a planepassing through the inferior margin of the left orbit and the uppermargin of each ear canal, which is parallel to the surface of the earth.

These procedures cause distress in the infant. Crying is common, and theinfant is likely to struggle, creating difficulties for personnelengaged in positioning the infant. Accurate and reliable measurementsare difficult to obtain. Due to these and other complications, obtainingfrequent (e.g., daily) measures of length is onerous and is not oftentried or accomplished.

The use of photos can offer an alternative that is not cumbersome andinvasive. In an embodiment, a photograph can be taken with a device thatcan provide a 3D digital image from one or more photos (i.e., asnapshot(s)). For example, a device that includes a time-of-flightcamera can provide the 3D digital image. Another example is use of astereo camera and structured light. In an embodiment, the device can bea cell phone, a mobile device, a computer, a laptop, a tablet or thelike that comprises a 3D imaging capability.

Most parents have a cell phone with a camera and parents enjoy takingsnapshot photos of their children, especially in infancy. Embodimentsdisclosed herein do not require precise positioning of the infant, anddo not because the distress associated with the infantometer procedures.Photos can be taken at standard times (upon awakening, at mealtime,after changing diapers, etc.) that can provide daily prompts forparents. The digital images from a 3D photo, or multiple 2D photosacquired from the parent's cell phone, for example, can be processed toprovide lengths between the defined anatomical indicia generated from aphoto or photos taken at a first time and compared to the lengthsbetween the anatomical indicia generated from a photo of photos taken ata second time, such as the next day or the next week, for example. Themultiple comparisons occurring over a period of time can provide anindication of the infant's growth.

There is a need to obtain a 3D image adequate to measure the length or apartial length of an infant (a) without specific equipment like aninfantometer, (b) without trained personnel like the two-person teamthat uses an infantometer, (c) without touching the infant and causingdistress while aligning the head and straightening and positioning thelegs and feet, (d) without the variability associated with a personplacing artificial landmarks on the infant each time an image is taken,and (e) without the use of multiple photos from different perspectives.A system and method that can be implemented easily in the home settingby parents to facilitate frequent and easy measurement of infant lengthis desirable.

FIG. 1A illustrates a block diagram of an example system 100 to acquireand process 3D digital images of a subject 104 to provide an estimate ofat least partial length of the subject 104, according to certainembodiments. The subject 104 can be an infant, child, adolescent, oradult. The illustrated system 100 comprises a mat 102, a 3D imageacquisition device 106, and an image processing and length estimationsystem 116. The 3D image acquisition device 106 and the image processingand length estimation system 116 communicate via a network 126. In someembodiments, the system 100 may include more or fewer components thanthose illustrated in FIG. 1A.

Network 126 may be any wired network, wireless network, or combinationthereof. In addition, network 126 may be a personal area network, localarea network, wide area network, cable network, fiber network, satellitenetwork, cellular telephone network, data network, or combinationthereof. In the example environment of FIG. 1A, network 126 is a globalarea network (GAN), such as the Internet. Protocols and components forcommunicating via the other aforementioned types of communicationnetworks are well known to those skilled in the art of computercommunications and thus, need not be described in more detail herein.

The mat 102 comprises a flat surface of known dimensions and withdefined corners. For example, the mat can be constructed of soft foam,e.g. yoga mat, and the like. The flat mat can be produced in severalsizes to accommodate a range of subject sizes. It is only required thatthe actual mat dimensions are known to the application software. It isalso required that the entire mat is included in the photograph of thesubject when the photograph is taken. In an embodiment, the corners ofthe mat 102 may be constructed with special marking, coloring, or otheridentification that makes the corners easy to recognize. The color ofthe remainder of the mat is chosen for low reflectivity of light at thefrequency of illumination. It is required that the back of subject is incontact with the mat in certain anatomical regions. This permits thealgorithm to compute a maximum depth measure for every point on thesurface of the subject. The mat 102 is an example of a fixed backgroundthat the subject 104 may be placed on or against. In other embodiments,other backgrounds may be used.

The 3D image acquisition device may be implemented as a single 106comprises at least processor(s) 108, a digital camera 110, memory 112,and a network interface 128 all of which may communicate with oneanother by way of a communication bus. In some embodiments, the 3D imageacquisition device 106 may include more or fewer components than thoseillustrated in FIG. 1A.

The network interface 128 may provide connectivity to one or morenetworks, such as network 126. The processor(s) 108 may thus transmitand receive information and instructions from other computing systemsvia a network. Processor(s) 108 may include one or more processors ormicroprocessors and may also communicate to and from memory 112.Processor(s) 108 may provide output information for an optional display(not illustrated) and may also accept input from an optional inputdevice (not illustrated), such as a keyboard, mouse, digital pen, touchscreen, etc.

The memory 112 may include computer program instructions that the 3Dimage acquisition device 106 executes in order to implement one or moreembodiments. The memory 112 generally includes RAM, ROM, or otherpersistent or non-transitory memory. The memory may store the 3D digitalimages 114 acquired by the digital camera 110. The memory 112 may alsostore an operating system that provides computer program instructionsfor use by the processor(s) 108 in the general administration andoperation of the 3D image acquisition device 106.

The digital camera 110 is configured to acquire 3D digital images. In anembodiment, the digital camera 110 includes a time-of-flight (TOF) 3Dimaging system which is used to acquire a single 3D image. In otherembodiments, multiple 2D images from multiple cameras could be used toacquire the 3D image. The TOF 3D imaging system acquires a singlepoint-of-view image by emitting infrared light or radio-frequencymodulated light and measuring the delay of the returning light as it isreflected off of the object. A camera lens gathers the reflected lightand images it onto a sensor, such as an image sensor comprising an arrayof pixels. Each pixel can be a photo sensitive element. An example of aphoto sensitive element is a photo diode. The delay and the intensity ofthe returning light are measured for an array of pixels. Each pixel inthe array has an amplitude value associated with the intensity thatprovides brightness information in addition to a value of the distancefrom the camera to the subject. In the TOF camera the distancecorresponds to a phase delay of the frequency modulated illumination andgiven the constant speed this can be converted to a distance value foreach pixel of light. The values can be stored in memory 112. The TOF 3Dimaging system may include a TOF component, such as, for example, aTI-OPT8241® by Texas Instruments, Inc., which includes a 320×240 arrayof pixels. In other embodiments, the array of pixels of the TOF 3Dimaging system may include more or fewer pixels.

In an embodiment, the 3D image acquisition device 106 comprises a cellphone, a mobile device, a computer, a laptop, a tablet or the like thatcomprises a time-of-flight camera 110 or a time-of-flight array. Forexample, the ZenFone AR® by ASUS is a commercially available cell phonethat includes a 3D TOF camera for 3D imaging.

In an embodiment, the image processing and length estimation system 116receives, via network 126, the 3D image information associated with thesnapshot of the subject 104 on the mat 102 that is acquired by theimaging acquisition device 106. For example, the 3D image informationincludes at least the intensity and distance values generated by thearray of pixels. The 3D image information may be referred to as the 3Ddigital image. The image processing and length estimation system 116 canprocess the 3D image information to generate a point-cloudrepresentation of the subject 104. The point-cloud is a set of datapoints in a coordinate system and can represent the external surface ofan object. The point-cloud representation of the subject 104 is a set ofcoordinates that represent the external surface of the subject 104 andis derived from the 3D image information of the subject 104. The imageprocessing and length estimation system 116 can further process thepoint-cloud representation to generate a pseudo-skeleton of the subject104. The pseudo-skeleton can also be referred to as a virtual skeleton.The pseudo-skeleton or virtual skeleton represents an approximation ofat least a portion of the skeleton of the subject 104 and is derivedfrom the point-cloud representation of the subject 104.

The illustrated image processing and length estimation system 116comprises at least processor(s) 118, memory 120, display 124, andnetwork interface 138 all of which may communicate with one another byway of a communication bus. In some embodiments, the image processingand length estimation system 116 may include more or fewer componentsthan those illustrated in FIG. 1A. Some components of the imageprocessing and length estimation system 116 may be physical hardwarecomponents or implemented in a virtualized environment.

The network interface 138 may provide connectivity to one or morenetworks, such as network 126. The processor(s) 118 may thus transmitand receive information and instructions from other computing systemsvia a network. Processor(s) 138 may include one or more processors ormicroprocessors and may also communicate to and from memory 120.Processor(s) 118 may provide output information for the display 124 andmay also accept input from an optional input device (not illustrated),such as a keyboard, mouse, digital pen, etc. The display 124 may be usedto display the various stages of the image processing, point-cloudrepresentations, and pseudo-skeletons.

The memory 120 generally includes RAM, ROM, or other persistent ornon-transitory memory. The memory 120 may include computer programinstructions that the image processing and length estimation system 116executes in order to implement one or more aspects of the presentdisclosure. For example, in one embodiment, the memory 120 may includean image processing component 128 comprising computer executableinstructions that when executed by the processor(s) 118 cause the imageprocessing and length estimation system 116 to process the 3D digitalimage to generate a point-cloud; a point-cloud processing component 130comprising computer executable instructions that when executed by theprocessor(s) 118 cause the image processing and length estimation system116 to process the point-cloud to generate a pseudo-skeleton; and agrowth determination component 132 comprising computer executableinstructions that when executed by the processor(s) 118 cause the imageprocessing and length estimation system 116 to determine an indicationgrowth from a plurality of pseudo-skeletons generated over a period oftime.

In addition, memory 120 may store digital images 114. The stored digitalimages may include the 3D digital images from the 3D image acquisitiondevice 106, various stages of the image processing, point-cloudrepresentations, pseudo-skeletons, and the like. The memory 120 may alsostore an operating system that provides computer program instructionsfor use by the processor(s) 118 in the general administration andoperation of the image processing and length estimation system 116.

FIG. 1B illustrates a block diagram of another example system 150 toacquire and process 3D digital images of the subject 104 to provide anestimate of at least partial length of the subject 104, according tocertain embodiments. The illustrated system 150 comprises the mat 102and an image acquisition device 106A. The image acquisition device 106Acomprises processor(s) 158, display 156, the digital camera 110, andmemory 152. In some embodiments, the system 150 may include more orfewer components than those illustrated in FIG. 1B.

Processor(s) 158 are similar to processor(s) 108 and 118 described aboveand display 156 is similar to display 124 described above. Furthermemory 152 is similar to memory 112 and 120 described above and caninclude the digital images 114, the imaging processing component 128,the point-cloud processing component 130, and the growth determinationcomponent 132. For example, functionality associated with processor(s)108 and 118 can also be provided by processor(s) 158, functionalityassociated with memory 112 and 120 can also be provided by memory 152,and functionality associated with display 124 can also be provided bydisplay 156.

In an embodiment, the image acquisition device 106A is a single devicethat includes the functionality of the image acquisition device 106 andthe functionality of the image processing and length estimation system116. For example, the image acquisition device 106A can comprise a cellphone or other mobile device that includes the TOF camera, processingfunctionality, and programming to be able to acquire the 3D digitalimage of the subject 104 on the mat 102, process the 3D digital image,generate the point-cloud representation, generate the pseudo-skeleton,and determine an estimate of at least the partial length of the subject104 based on the pseudo-skeleton.

Referring to FIG. 1A, a user, such as a parent, caregiver, clinician orthe like, for example, positions the image acquisition device 106 aboveor in front of the mat 102 having known dimensions. The mat 102 is alsoreferred to as the floor or wall. The user uses the image acquisitiondevice 106 to take a snapshot of the mat 102. The image acquisitiondevice stores the 3D digital image of the mat 102 in the memory 112. Inan aspect, the mat 102 is photographed such that the entire mat 102 iscaptured in the snapshot and in the resulting 3D digital image of themat 102.

The user also takes a snapshot of the subject 104 on the mat 102. Theimage acquisition device 106 stores a 3D digital image of the subject104 on the mat 102 in memory 112. In one embodiment, the subject 104 isplaced on the mat 102 in a supine position (lying on one's back), and inanother embodiment in front of the mat (standing upright). In someembodiments, the back of the subject's body touches the mat 102 and thesubject's arms are at the sides of the body or at a position that doesnot obscure the subject's torso. Advantageously, the subject is notrequired to assume an abnormal or uncomfortable posture. Because thesubject is not forced into an abnormal or uncomfortable position, theabstract representation of the subject is more replicable and accuratethan an abstract representation of the subject acquired by forcing thesubject into an abnormal posture.

In an aspect, the subject 104 on the mat 102 is photographed such thatthe entire mat 102 and the entire subject 104 are captured in thesnapshot and in the resulting 3D digital image of the subject 104 on themat 102. In an embodiment, the body of the subject fills the availableframe as much as possible for maximum resolution. In an embodiment, theknown height of the digital camera 110 above the subject 104 isapproximately at a height of twice the length of the subject 104. Lensoptics could be chosen to optimize parameters of distortion versusresolution.

Turning now to FIG. 2 , a routine 200 utilized by the image processingand length estimation system 116 to generate a point-cloudrepresentation of subject using at least two 3D snapshots will bedescribed. Computer algorithms filter and transform the 3D digital imageassociated with the snapshots of the background and the image of thesubject on the background to form a 3D image of the subject. In anembodiment, the transforming modifies the image and filtering smoothsthe image. In another embodiment, filtering includes determiningmathematical derivatives in length and width planes of the image tolocate peaks and troughs.

At step 202, the image processing and length estimation system 116acquires the 3D digital image of the background, such as the mat 102. Asdescribed above, the mat 102 is an example of a fixed background thatthe subject 104 may be placed on or against. For example, the imageacquisition device 106 sends the image information associated with thepixel array for the mat 102 to the image processing and lengthestimation system 116 via the network 126. The 3D digital image of themat 102 without the subject 104 can be used to calibrate measurements inthe X, Y, and Z dimensions. The image of the background, such as the mat102, may be reused.

The following convention for labeling the three-dimensional form isused: the Y-axis runs from head to toe, the X-axis runs from shoulder toshoulder, and the Z-axis is represented as a distance from the surfaceof the subject to the to the mat 102, which is also referred to as thefloor, for example.

Since the mat is known to be temporally stable and approximatelyspatially flat, this can be sufficient information to interpolate anypoint on the floor or wall in three dimensions to desired precision. The3D digital image of the mat 102 and the known geometry of the scene canthen be used in one or more subsequent images to measure the desireddistance from each point on the surface of the subject 104 to thenearest point on the floor, along a line perpendicular to the floor.

At step 204, the image processing and length estimation system know thedimensions of the mat and computes a calibration transform that canrotate the entire 3D image such that the camera now accuratelypositioned on a line perpendicular to the center of the mat 102. Aftertransformation The X and Y scales for the subject surface are then alsoadjusted to centimeters. After application of this transformation to thecomposite image, where the composite image is the 3D digital image ofthe subject 104 on the mat 102, every point in the composite image wherethe mat 102 is visible also becomes nearly the same constant value. Whenthe two transformed images are then subtracted, the distance values forthe mat 102 become nearly zero and all points on the surface of thesubject 104 are some positive value of height above the floor. Theresulting image is a point-cloud representation of the subject 104. Thispoint-cloud representation is explored for anatomical features.

At step 206, the image processing and length estimation system 116acquires the 3D digital image of the subject 104 on the mat 102. Forexample, the image acquisition device 106 sends the image informationassociated with the pixel array for the subject 104 on the mat 102 tothe image processing and length estimation system 116 via the network126.

FIG. 2A is an example of an image of the subject 104 on the mat 102acquired by the image processing and length estimation system 116 fromthe image acquisition device 106.

At step 208, the image processing and length estimation system 116applies the calibration transformation that was applied to the 3Ddigital image of the mat 102 to the 3D digital image of the subject 104on the mat 102. As described above, after application of the calibrationtransformation to the 3D digital image of the subject 104 on the mat102, points in the transformed image of the subject 104 on the mat 102where the mat 102 is visible also become nearly the same constant value.In another embodiment, a different calibration transformation is appliedto the 3D digital image of the subject 104 on the mat 102 than isapplied to the 3D digital image of the mat 102.

At step 210, the image processing and length estimation system 116transforms the image data by subtracting the transformed image data forthe mat 102 from the transformed image data for the subject 104 on themat 102.

In another embodiment, the 3D digital image of the mat 102 is acquiredonce, transformed, and stored in memory 120. Routine 200 subtracts thesame stored, transformed image of the mat 102 from the transformed imagedata for the subject 104 on the mat 102 for multiple 3D digital imagesof the subject 104 on the mat 102 acquired over time.

At step 212, the image processing and length estimation system 116obtains the distance of each point of the surface of the subject 104 tothe floor from the subtraction. When the two images are subtracted, thedistance values for the mat 102 become nearly zero and points on thesurface of the subject are positive values with respect to the floor.

FIG. 2B is an example transformed image 630 of the subject 104represented by the distance of each point of the surface of the subjectto the floor. Image 630 represents a point-cloud describing the surfaceof the subject's body. The points of the point-cloud are vertices of aconvex mesh restricted to a two-dimensional hypersurface. Each vertex isreferenced to a corresponding point on the floor. As described infurther detail below, anatomical features or natural landmarks of thesubject 104 are placed on the point-cloud representation of the subject104.

Turning now to FIG. 3 , a routine 300 utilized by the image processingand length estimation system 116 to generate a point-cloudrepresentation of an subject using a single snapshot will be described.Taking a single snapshot of the subject 104 on the mat 102, instead ofalso needing to photograph the mat 102 alone, makes it much easier andconvenient for parents or other caregivers to photograph the subjectfrequently, such as daily. As described above, the mat 102 is an exampleof a fixed background that the subject 104 may be placed on or against.

The user, such as the parent or other caregiver takes a snapshot of thesubject 104 on the mat 102 without taking a snapshot of the mat 102alone. The snapshot of the subject 104 on the mat 102 results in a 3Ddigital image of the subject 104 on the mat 102 stored in memory 112. Asdescribed above, the subject 104 can be placed on the mat 102 in asupine position, with the back of the subject's body touching the mat102 and the subject's arms laying the sides of the body or at a positionthat does not obscure the subject's torso. Advantageously, the subjectis not required to assume an abnormal or uncomfortable posture. Becausethe subject is not forced into an abnormal or uncomfortable position,the abstract representation of the subject is more replicable andaccurate than an abstract representation of the subject acquired byforcing the subject into an abnormal posture.

At step 302, the image processing and length estimation system 116acquires the distance values associated with depth information for the3D digital image of the subject 104 on the mat 102. In an embodiment,the image processing and length estimation system 116 acquires thedistance values from the image acquisition device 106, via network 126.Mat 102 has known dimensions and observable corners.

At step 304, the image processing and length estimation system 116acquires the amplitude values associated with brightness information ofthe reflected light for the 3D digital image by the subject 104 on themat 102. In an embodiment, the image processing and length estimationsystem 116 acquires the amplitude values from the image acquisitiondevice 106, via network 126.

For example, the TOF camera 110 stores an amplitude value that providesbrightness information associated with the returning light and adistance value that provides depth information for each snapshot. Theimage acquisition device 106 can send the amplitude and distance valuesassociated with the pixel array for subject 104 on the mat 102 snapshotto the image processing and length estimation system 116 via the network126. The array of amplitude values can be characterized as an amplitudeimage associated with brightness and the array of distance values can becharacterized as a depth image.

FIG. 3A is an example 3D depth image 710 of the subject 104 on the mat102. The 3D depth image 710 can also be referred to as the phase image710. When the 3D depth image 710 is rotated to horizontal, asillustrated by example image 730 in FIG. 3B, the height of the points onthe surface of the subject 104 above the floor are shown. FIG. 3C is anexample amplitude image 720 of the subject 104 on the mat 102. Theamplitude image 720 can also be referred to as the intensity image 720.The each Z dimension along the Z-axis in the amplitude image 720represents the brightness of the corresponding pixel in the pixel array.

At step 306, the image processing and length estimation system 116 usesthe amplitude image 720 to remove at least a portion of the noise and/orreflections from the depth image 710. This is accomplished by detectingand replacing outlier pixels and smoothing with a Gaussian kernel.

At step 308, the image processing and length estimation system 116applies one or more smoothing algorithms to the depth image 710 togenerate one or more secondary images. In an embodiment, the imageprocessing and length estimation system 116 applies one or more spatialfilters to the depth image 710 to generate the secondary images. In anembodiment, the secondary images are different versions of the depthimage 710, where each version has a different amount of smoothing orfiltering. The image processing and length estimation system 116 canstore the secondary images in the memory 120. The smoothing filters wereGaussian and in addition a single median filter was used to replaceoutlier pixels in the real images.

Calibration information can be derived from the edges of the mat 102. Atstep 310, the image processing and length estimation system 116 canrotate the view perspective of the secondary images so that resultingrotated images represent the digital camera 110 appearing in virtualspace directly above the mat 102. For example, the image processing andlength estimation system 116 can detect the mat edges. In an embodiment,the corners of the mat 102 are detected. After detecting the mat edges,the image processing and length estimation system 116 can rotate thesecondary images to flat and rectangular, and can scale the secondaryimages to the known dimensions of the mat 102.

The virtual plane of the digital camera 110 is aligned with the mat 102in the rotated images. In the rotated secondary images, theapproximately all of the depth values for the mat 102 are approximatelya constant value plus noise. This value is then subtracted from thecomposite 3D image.

FIGS. 3D and 3E are example transformed images of the subjectrepresenting the surface of the subject's body. FIG. 3D illustrates anexample image 740 with detected mat corners that has been rotated to beflat and rectangular and scaled according to the known mat dimensions.FIG. 3E illustrates an example image 750 of the flat, rectangular, andscaled image 740 visualized along the Y-axis. Horizontal image 750 showsthe depth information along the Z-axis.

At step 312, the image processing and length estimation system 116estimates the noise threshold of the noise. This noise threshold isderived from the variation of background which now is variation aboutzero. For example, if this is 1 cm, then all points with Z values lowerthan 1 cm are removed in the original surface leaving only points on thefrontal surface of the subject.

At step 314, the image processing and length estimation system 116process the secondary images by removing the mat 102 from the rotatedsecondary images. The image processing and length estimation system 116removes the mat 102 from the rotated secondary images by deleting thepoints in the secondary images having a Z (height) value that is lessthan the estimated noise threshold. In an embodiment, the imageprocessing and length estimation system 116 removes the mat 102 from therotated secondary images by deleting the points in the secondary imageshaving a Z value that is less than the constant value plus estimatednoise threshold.

After rotation, there is no longer a fixed relationship between thecamera pixels of the acquired 3D digital image and the points of therotated secondary images. In an embodiment, the points of the rotatedsecondary images represent vertices of a mesh over a surface of thesubject.

At step 316, the image processing and length estimation system 116combines data from one or more of the processed secondary images to forma point-cloud representation of the subject 104. Data from the processedsecondary images are combined to provide multiple Z values for thepoints on the point-cloud representation having auxiliary informationabout brightness and surface curvature. The smoothing filters wereoptimized based on the relative size of the different anatomicallandmarks.

FIG. 3F illustrates an example image 760 of the point-cloudrepresentation of the subject 104. In an embodiment, the point-cloudrepresentation is a noise free or approximately noise free mesh surfaceapproximation of the subject 104.

Turning now to FIG. 4 , a routine 400 to estimate the length or partiallength of the subject from a generated pseudo-skeleton of the subject104 will be discussed. The estimation of length begins with thecomputation of the principal axes of the points on the surface image ofthe subject 104. As described above, the following convention forlabeling the three-dimensional form is used: the Y-axis runs from headto toe, the X-axis runs from shoulder to shoulder, and the Z-axis isrepresented as a distance from the surface of the subject to the floor.

At step 402, the image processing and length estimation system 116computes the principal axes of the point-cloud. In an embodiment, theimage processing and length estimation system 116 further rotates thepoint-cloud representation to vertical. This provides a definition ofthe principal axes of the point-cloud representation of the subject 104.The longest axis is orientated through the volume of the subject surfaceso that the representation of the subject 104 is aligned with the Y-axisof the measurement frame. The point (0, 0, and 0) is defined as Z₀. Inan embodiment, Z₀ is located at the intersection of the Y-axis with thelongest axis across the volume of the subject surface (defined as theX-axis).

FIG. 4A illustrates an example cloud point representation 820 showingthe Y-axis, the X-axis, and Z₀. In the example point-cloudrepresentation, the X-axis is located approximately mid-way along thetorso of the subject 104.

At step 404, the image processing and length estimation system 116 scansthe point-cloud representation along the Y-axis to provide physicalmeasurements along the length. Further, the image processing and lengthestimation system 116 filters the point-cloud representation to refinethe image and to locate peaks and troughs. FIGS. 3D and 3E illustrateexamples of filtered images 740 and 750, respectively. Several differentGaussian smoothing spatial integration used by filters assist detectionof known anatomical features i.e. the peak of the forehead by smoothingthe head so that a clear convex surface is always produced even in thepresence of a small spot or ridge.

At step 406, the image processing and length estimation system 116processes the filtered image to locate natural landmarks of thesubject's anatomy on the image. In an aspect, the image processing andlength estimation system 116 locates target regions of the naturallandmarks using heuristic processes and statistics of the physicalmeasurements. Examples of natural landmarks include, but are not limitedto the eyes, navel, neck, knees, hip, ankle, heel, top of head, bottomof feet, arms, wrist, elbow, shoulder, centroid of the head, and thelike.

Advantageously, the use of natural landmarks can be reliably replicatedin subsequent images of the subject 104 and do not require the sameposture of the subject 104 for each image. This is different from theneed to measure absolute height (i.e. from the top of the head to theheel in a standing or supine position), which requires the same posturefor each observation. Because the natural landmarks can be reliablyreplicated in subsequent images, and can be used to create thepseudo-skeleton of the infant, indications of growth can be determinedby the change in the length between the natural landmarks as images areacquired over time (if they are independent of postural variation). Forexample, the distance from a point midway between the eyes and theaverage of the distance to each knee can be determined in each of aseries of 3D images taken over time. As the infant grows, the increasein this distance (or partial length) is proportional to the overallgrowth of the subject. In other words, the partial length determined bythe difference of the distance between first and second naturallandmarks can be compared for the observation acquired at a first pointin time to the partial length (the distance between the same first andsecond natural landmarks) acquired at a later point in time, and thechange on the partial length provides an indication of the growth of thesubject over the time between the two observations.

The use of natural landmarks avoids the use of indeterminate position oflocations. The top of the head that may have more or less hair andirregular form, making the exact location of the top of the headdifficult to determine. Further, soft and flexible anatomy, such as thebottoms of the feet, may result in non-repeatable or ambiguousmeasurements. A model head or model feet may be generated used to locatethe positions that could be used along with the pseudo-skeleton toobtain the absolute length without having to position the subject byaligning the head and stretching the legs.

At step 408, the image processing and length estimation system 116estimates the initial size and location of each natural landmark. In anembodiment, the image processing and length estimation system 116 uses afeature detection algorithm to compute a general estimate of size orvolume of the filtered image. The estimate of the size or volume canprovide calibration information for detecting features of the subject104 on the point-cloud representation. For example, the estimated volumepermits the image processing and length estimation system 116 toestimate the relative volume and curvature for each class of anatomicalfeatures or natural landmarks that are used to estimate the skeletalstructure of the subject 104. Volume estimates are possible because thealgorithm actually does produce a closed volume with the frontal surfacereflecting the true surface of the subject. Although the back surface isassumed flat, there is not a lot of residual volume in this method.Although it is always somewhat larger than the true volume, thisestimate will replicate well for repeated measurements.

Random error is reduced in the algorithm overall because of the methodused to fit spherical point-clouds to the interior of this volume. Ageneral knowledge of anatomy allows us to select a region of the body asshown in FIG. 6 a , 6B that would contain a joint. This is viewed aspoint-cloud band around the torso and each leg as a general constraintover the length. A sphere is created with a radius estimating theinterior volume. This is also guide by the heuristic component thatknows the knees may be up in the air but assumes hips are on the floor.Each sphere is then snugly fit as closely as possible in the interior ofthe surface mesh using a simplified point set regression.

Once the feature detection algorithm estimates the general surface ofthe physical form, the feature algorithm can distinguish landmarks alongits length like the bifurcation of the legs at one end of the torso.Successive filters with various kernels are applied to the 3D data ofpoints on the body that are optimized for anatomical features andnatural landmarks, such as, but not limited to:

a. Head, Head

b. Neck, Chest

c. Hips, Thighs

d. Knees, Calves

e. Ankles, Feet

In an embodiment, methods using one or more heuristic algorithms can beused to identify the natural landmarks. An embodiment of a heuristicalgorithm may comprises the following measurements:

-   -   i. Detect the orientation of the longest axis of the subject's        body, Y axis;    -   ii. Scanning along this axis all points within a specified range        of Y coordinates define a band of points around the torso at        every multiple of some length resolution (for example, a        parameter set to 1% of length).    -   iii. Each band is comprised of a number of points on the upper        surface of the subject;    -   iv. Each band also contains an equal number of points on the        inferior surface with the same X and Y values and a Z value of        zero.    -   v. Hence, there is now a closed volume defined at each segment        of length and it can be associated with a number of measures:        -   a. Volume of segment        -   b. The minimum and maximum range of X, Y, Z.        -   c. The associated centroids center of mass.        -   d. In the X axis or width, sub-centroids are computed of the            remaining volumes on left and right side of the center.

In an embodiment, methods using the measurements described above and aheuristic algorithm can be used to identify natural landmarks as extremaand points of articulation along the length of the subject. The firstheuristic may assume the range for identification of all features forlength computation lies between the first and last segments definedabove. These landmarks may be accurately estimated as follows:

-   -   i. NECK: The neck segment is identified as the minimum volume        segment. This also identifies which end of the body contains the        head.    -   ii. FOREHEAD: Scanning now from the first segment, the forehead        segment is the first volume maximum.    -   iii. HIPS: Continuing this scan, the first point of bifurcation,        the top of legs, is defined when Z maxima of sub-centroids        exceed Z maximum of total centroid. The hip segment is the last        maximum in width prior to this point. The X axis volume centroid        divides this segment into left and right hip segments.    -   iv. KNEES: Are detected as a variation in cross section

For example, there may not be a clear neck feature in a view of a chubbyinfant but a centroid of the diminished mass between head and torsowould be detectable and sufficient to locate the neck joint used for themodel. An estimate of overall length does not vary significantly if theneck is approximated by a single joint between head and torso ratherthan defined by the seven vertebrae of realistic anatomy.

At step 410, the image processing and length estimation system 116generates a sphere within a volume of each natural landmark. In anembodiment, the image processing and length estimation system 116generates the best fitting sphere, where the volume of the sphereclosely matches the volume of the natural landmark.

At step 412, the image processing and length estimation system 116connects vertices of the spheres to visualize pseudo-joints and extremaof a virtual skeleton on the image. The virtual skeleton can also bereferred to as a pseudo-skeleton. The spheres placed in the image aresized to correspond to the volume of the individual landmark. Forexample, the sphere located at the infant's hip is larger than thesphere located at the infant's knee. The set of pseudo-joints may differin number from image to image, but for a given 3D digital image therewill be enough to estimate the small number of vertices required for avirtual skeleton.

FIG. 4B illustrates an example of a computer-generated side view 910 ofa virtual skeleton. FIG. 4C illustrates an example of acomputer-generated top view 920 of a virtual skeleton. The views 910,920 of the virtual skeleton indicate the locations of the eyes, neck,pelvic joints, and knees.

FIG. 4D illustrates an example of a side view 930 of a 3D image of thesubject 104 including the virtual skeleton of FIG. 4B. FIG. 4Eillustrates an example of a top view 940 of a 3D image of the subject104 including the virtual skeleton of FIG. 4C. FIGS. 4D and 4E furtherillustrate the depth information provided by a single frame from a TOFcamera.

FIGS. 4F and 4G illustrate other examples of computer-generated virtualskeletons 1000, 1010. FIG. 4F illustrates a top-view 1000 and FIG. 4Gillustrates a side-view 1010.

At step 414, the image processing and length estimation system 116estimates the length of the subject as the distances between the centersof the spheres for selected portions of the virtual skeleton. Thecumulative distances along the Y-axis, from one pseudo-joint to thenext, provides an estimate of the length of the subject 104. In someembodiments, the image processing and length estimation system 116determines the total overall length of the subject 104. In otherembodiments, the image processing and length estimation system 116determines a partial length of the subject 104. For example, thedistance between the neck and the hips, the distance between the hipsand the knees, the distance between the neck and the knees, or any otherpartial length of the subject 104 or any combination of partial lengthscan be used to provide an indication of the growth of the subject 104.

Turning now to FIG. 5 , a routine 500 to provide indication of growthwill be described. At step 502, the image processing and lengthestimation system 116 acquires and processes the 3D digital images ofthe subject 104 on the mat 102. The 3D digital images can be receivedperiodically. In an embodiment, the period between receiving consecutive3D digital images is short so as to distinguish the infant's growthbetween saltations and stasis. Examples of short periods betweenreceiving consecutive 3D digital images of the subject 104 on the mat102 are daily, weekly, monthly, hourly, every 12 hours, 4 times a day,twice a day, and the like. In an embodiment, the 3D digital images areprocessed according to embodiments described with respect to FIG. 2 . Inanother embodiment, the 3D digital images are processed according toembodiments described with respect to FIG. 3 . The image processing andlength estimation system 116 further generates virtual skeletons fromthe processed images. In an embodiment, the virtual skeleton isgenerated according to embodiments described with respect to FIG. 4 .

At step 504, the image processing and length estimation system 116determines estimates of length or estimates of partial length bymeasuring the distances between the spheres in one or more portions ofthe virtual skeletons for each processed image. The cumulative distancesbetween the spheres in the one or more portions of the virtual skeletonsprovides estimates of the subject's length or partial length.

At step 506, the image processing and length estimation system 116stores the estimates of the subject's length or partial length.

At step 508, the image processing and length estimation system 116compares the cumulative distances from the processed images (i.e., theestimates of the subject's length or partial length) to determine achange in the subject's length or partial length over time. For example,a first estimate of subject length from a first 3D digital image iscompared to a next estimate of subject length from a next 3D digitalimage, taken later in time than for first 3D digital image. The changein estimated subject length between successive measurements over timeprovides an indication of growth. The indication of growth may becompared with normal growth to determine if the subject's growth isnormal and to provide timely intervention when the subject's growth isfaltering.

FIG. 6 illustrates an example of a prototype TOF 3D imaging system 1100to capture a single 3D image of the subject 104. The TOF system 1100includes a TOF device and a computer. The TOF device emits modulatedlight and measures the time-of-flight of the reflected light from eachpoint of the surface of a 3D object that is facing the camera. The TOFdevice may include a TOF component, such as a TI-OPT8241® by TexasInstruments, Inc. In other embodiments, devices that include a camera,such as a digital camera, a tablet, a laptop computer, and the like, canbe used to acquire the 3D image. In an embodiment, the included camerais a TOF camera.

The TOF 3D imaging system 1100 acquires a single point-of-view image byemitting infrared light and measuring the delay of the returning lightas it is reflected off of the subject to estimate distance. The delay ismeasured with an array of sensors. In the prototype embodimentillustrated in FIG. 6 , a 320×240 array of sensors was used. In otherembodiments, the array of sensors comprises more or less sensors.

The time-of-flight device can be implemented on a cell phone, such asthe ZenFone AR® by ASUS, but could be implemented on other wirelessdevices, such as laptop computers, digital cameras, and the like. Thiswill provide an easy and convenient way to measure length of infantsfrom frequent, daily photographs provided by infant caregivers in thehome setting. Thus, the time-of-flight method offers a way to obtain anestimate of length of an infant from a single 3D image acquired withouttrained personnel, and using easily available equipment. FIGS. 7A and 7Billustrates the front and back views, respectively, of a commerciallyavailable mobile device, the ZenFone AR® by ASUS, which includes a TOFcamera.

In other embodiments, the 3D digital image may be obtained fromsynchronized (i.e., phased) multi-camera TOF systems. To mitigateinterference between the images acquired by each TOF camera in themulti-camera TOF system, each TOF camera is paired with a light source,and each light source is operated at a different frequency. In a furtherembodiment, the 3D digital image may be obtained from a single TOFcamera receiving light reflected from the photographed object that isilluminated by multiple light sources, each light source operating at adifferent frequency.

In addition to TOF, there are other systems and methods to acquire a 3Ddigital image of the subject that can be processed as described above todetermine the change in length over time of the distance on a virtualskeleton between selected natural landmarks.

FIG. 8 illustrates an example embodiment of a structured light system1200 used to capture the 3D digital image of the subject 104. Astructured-light 3D imaging system includes a scanning device formeasuring the three-dimensional shape of an object using projected lightpatterns and a dual camera system. Projecting a narrow band of lightonto a three-dimensionally shaped surface produces a line ofillumination that appears distorted from other perspectives than that ofthe projector, and can be used for a geometric reconstruction of thesurface shape.

The structured light system 1200 may comprise a structured-light device,such as Structure Sensor®, attached to an iPhone®. The structured-light3D imaging system acquires multiple point-of-view images by moving thestructured-light device around the subject and, using an imageaggregating processor such as the processor in the iPhone® or acomputer, dynamically aggregates point-clouds, such as an array ofpixels acquired by the structured-light device, to form a 3D digitalimage of the subject.

In another embodiment, the mat 102 can comprise a clear material, suchas a clear acrylic, Plexiglas®, and the like, and images from cameraslocated below the clear mat can be used to provide information on theback of the subject. This information from the bottom-view can be usedto increase accuracy of the position of points on the surface of thesubject and improve estimates of length based on the natural landmarkson the front of the subject in the 3D digital image from just thetop-view.

Other 3D imaging systems and methods that could be used to acquire 3Ddigital images are, for example, but not limited to, applying principlesof photogrammetry to a standard cell phone camera, other devices using astereo camera (e.g. commercially available on the Leap®, Lucid®,Blackbird® devices, and the like), and some cell phones with 3D cameras(e.g., ZenFone AR).

These systems provide 3D digital images that can be processed asdescribed above.

Other Embodiments

In an embodiment, a device is embedded in an attachment to aninfantometer or scales for weighing an infant, both of which aretypically found in medical facilities. The device may comprise atime-of-flight 3D imaging system (such as the PMD CamBoard® system), astructured-light 3D imaging system, a stereo camera system, or the likeon an elongated structure, such as a stick (“pixastick”). A photo wouldbe taken before the infant is placed on the infantometer, and then againafter placement of the infant, but without alignment of the head orpositioning of the legs. The images could be processed using the methodsdescribed herein to locate landmarks and to measure the distance betweenthem to provide an indication of the infant's growth.

Weight information from the scale can be used with the estimate ofvolume from the processed image to provide an indication of the densityof the infant. Along with length, weight can be used to provide anestimate of body mass index (BMI), which is used and understood inauxology even though it has technical weaknesses. In an embodiment, thedensity of the infant is also good indicator of infant growth andoverall health.

In another embodiment, a pressure sensitive mat can be used to provideinformation on the position of critical points (i.e., the back of thehead, heel of the foot, etc.) from the back of the infant. The pressuresensitive mat can provide weight information. Sometimes, a bulky diapercan obscure the natural landmarks. The use of a pressure sensitive matcan provide additional information as to the location of the head, hips,and other natural landmarks, based on the pressure measurements when theinfant is placed on the pressure sensitive mat. This can overcome anychallenges presented by diapers and other clothing.

In another embodiment, a 3D imaging device, such as the time-of-flightimaging device or the like, is placed on an infantometer. With thealgorithm supplied in FIG. 1 . The device could be mounted on a stand soit is placed over the infant. Either continuously or manually, the 3Dimaging device would take images of the empty infantometer and theinfantometer with the infant in it, and transmits the images to astorage and processing device, such as a cell phone, computer, or thelike, for storage and processing of the series of images.

In another embodiment, multiple time-of-flight devices or cameras can beincorporated in an arrangement at the ends of the infantometer, as wellas above the infant. These could be used to obtain better resolution ofthe head and the feet, which could be added to the length of body partsthat are identified from the main image taken from above the infant.

In an embodiment, a two-camera approach to obtain a stereo image isexpanded taking multiple images from different perspectives. The camerascan be mounted on the elongated structure (“pixastick”) attached to aninfantometer. A commercial program, such as Photomodeler™, AutoDesk™,123D™, or the like, can generate 3D digital images from multiple photos.Once the 3D digital image is acquired, it can be processed as describedherein to provide the indication of infant growth.

In another embodiment, a commercially available box designed to providea safe sleeping place for an infant to prevent or reduce sudden deathwhile sleeping, such as the BabyBox™ and the like can have adevice/camera embedded in the four corners or elsewhere in knownlocations, or the “pixastick” be placed on top, and the multiple imagessnapped automatically or manually to provide information to generate a3D digital images. Once the 3D digital image is acquired, it can beprocessed as described herein to provide the indication of infantgrowth.

In a further embodiment, 3D images of a fetus generated by ultrasoundduring pregnancy can be processed as described herein, (filtered,transformed, and landmarks identified) and used to provide theindication of growth of the fetus during pregnancy.

Terminology

Depending on the embodiment, certain acts, events, or functions of anyof the algorithms described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of thealgorithm). Moreover, in certain embodiments, acts or events can beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

All of the processes and steps described above as being implemented bythe identification and marketing service may be performed and fullyautomated by a computer system. The computer system may include multipledistinct computers or computing devices (e.g., physical servers,workstations, storage arrays, etc.) that communicate and interoperateover a network to perform the described functions. Each such computingdevice typically includes a processor (or multiple processors) thatexecutes program instructions or modules stored in a memory or othernon-transitory computer-readable storage medium or device. The variousidentification and marketing service functions disclosed herein may beembodied in such program instructions, although some or all of thedisclosed functions may alternatively be implemented inapplication-specific circuitry (e.g., ASICs or FPGAs) of the computersystem. Where the computer system includes multiple computing devices,these devices may, but need not, be co-located. The results of thedisclosed methods and tasks may be persistently stored by transformingphysical storage devices, such as solid-state memory chips and/ormagnetic disks, into a different state.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Depending on the embodiment, certain acts, events, or functions of anyof the algorithms described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of thealgorithms). Moreover, in certain embodiments, acts or events can beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

Systems and modules described herein may comprise software, firmware,hardware, or any combinations of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser, or via other means suitable for thepurposes described herein. Data structures described herein may comprisecomputer files, variables, programming arrays, programming structures,or any electronic information storage schemes or methods, or anycombinations thereof, suitable for the purposes described herein. Userinterface elements described herein may comprise elements from graphicaluser interfaces, command line interfaces, and other suitable interfaces.

Further, the processing of the various components of the illustratedsystems can be distributed across multiple machines, networks, and othercomputing resources. In addition, two or more components of a system canbe combined into fewer components. Various components of the illustratedsystems can be implemented in one or more virtual machines, rather thanin dedicated computer hardware systems. Likewise, the data repositoriesshown can represent physical and/or logical data storage, including, forexample, storage area networks or other distributed storage systems.Moreover, in some embodiments the connections between the componentsshown represent possible paths of data flow, rather than actualconnections between hardware. While some examples of possibleconnections are shown, any of the subset of the components shown cancommunicate with any other subset of components in variousimplementations.

Embodiments are also described above with reference to flow chartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products. Each block of the flow chart illustrationsand/or block diagrams, and combinations of blocks in the flow chartillustrations and/or block diagrams, may be implemented by computerprogram instructions. Such instructions may be provided to a processorof a general purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the acts specified in the flow chart and/or block diagramblock or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flow chart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer or other programmable data processing apparatusto cause a series of operations to be performed on the computer or otherprogrammable apparatus to produce a computer implemented process suchthat the instructions which execute on the computer or otherprogrammable apparatus provide steps for implementing the acts specifiedin the flow chart and/or block diagram block or blocks.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the disclosure. Indeed, the novel methods and systems describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the describedmethods and systems may be made without departing from the spirit of thedisclosure.

What is claimed is:
 1. A computer-implemented method to estimate growthof a subject, the computer-implemented method comprising: accessing aplurality of three-dimensional image of a subject captured over a periodof time; generating respective virtual skeletons from the plurality ofthree-dimensional images of the subject; determining estimates of apartial length for respective virtual skeletons, generated from theplurality of three-dimensional images of the subject, by measuring adistance between landmarks of the virtual skeletons for each processedimage; comparing the estimates of partial lengths to determine a changein the subject's partial length over time and to provide an indicationof growth; and comparing the indication of growth with a referencegrowth to determine an advisability of an intervention.
 2. Thecomputer-implemented method of claim 1, wherein the landmarks compriseat least one point of inflection.
 3. The computer-implemented method ofclaim 1, the method further comprising: processing the plurality ofthree-dimensional images of the subject to generate respectivepoint-cloud representations of the subject; and using the point-cloudrepresentations of the subject to generate respective virtual skeletons.4. The computer-implemented method of claim 1, wherein a first of thethree-dimensional images of the subject is captured using a time offlight device.
 5. The computer-implemented method of claim 1, wherein afirst of the three-dimensional images of the subject is captured using amobile communication device camera.
 6. The computer-implemented methodof claim 1, wherein a first of the three-dimensional images of thesubject and a second of the three-dimensional images of the subject arecaptured within a week of each other.
 7. The computer-implemented methodof claim 1 wherein a first of the three-dimensional images of thesubject comprises an image of the subject and a background of knowndimensions, the method further comprising: acquiring distance valuesassociated with depth information for the first of the three dimensionalimages of the subject on the background; and acquiring amplitude valuesassociated with brightness information of reflected light for the firstof the three dimensional images of the subject on the background.
 8. Asystem comprising: one or more computing devices associated with ameasurement system, wherein the measurement system is configured to:access a plurality of three-dimensional image of a subject captured overa period of time; generate respective virtual skeletons from theplurality of three-dimensional images of the subject; determineestimates of a partial length for respective virtual skeletons,generated from the plurality of three-dimensional images of the subject,by measuring a distance between landmarks of the virtual skeletons foreach processed image; use the estimates of partial lengths to determinea change in the subject's partial length over time and to provide anindication of growth; and use the indication of growth and a referencegrowth to determine an advisability of an intervention.
 9. The system ofclaim 8, wherein the landmarks comprise at least one point ofinflection.
 10. The system of claim 8, wherein the system is furtherconfigured to: process the plurality of three-dimensional images of thesubject to generate respective point-cloud representations of thesubject; and use the point-cloud representations of the subject togenerate respective virtual skeletons.
 11. The system of claim 8,wherein a first of the three-dimensional images of the subject iscaptured using a time of flight device.
 12. The system of claim 8,wherein a first of the three-dimensional images of the subject iscaptured using a mobile communication device camera.
 13. The system ofclaim 8, wherein a first of the three-dimensional images of the subjectand a second of the three-dimensional images of the subject are capturedwithin a week of each other.
 14. The system of claim 8, wherein a firstof the three-dimensional images of the subject comprises an image of thesubject and a background of known dimensions, the system furtherconfigured to: acquire distance values associated with depth informationfor the first of the three dimensional images of the subject on thebackground; and acquire amplitude values associated with brightnessinformation of reflected light for the first of the three dimensionalimages of the subject on the background.
 15. Computer-readable mediathat stores instructions, that when executed by a computing device causethe computing device to perform operations comprising: access a firstplurality of three-dimensional image of a subject captured over a periodof time; generate respective virtual skeletons from the first pluralityof three-dimensional images of the subject; determine estimates of atleast a partial length for respective virtual skeletons, generated fromthe plurality of three-dimensional images of the subject, by measuring adistance between landmarks of the virtual skeletons for each processedimage; use the estimates of partial lengths to determine a change in thesubject's partial length over time and to provide an indication ofgrowth; and use the indication of growth and a reference growth todetermine an advisability of an intervention.
 16. The computer-readablemedia of claim 15, wherein the landmarks comprise at least one point ofinflection.
 17. The computer-readable media of claim 15, the operationsfurther comprising: process the first plurality of three-dimensionalimages of the subject to generate respective point-cloud representationsof the subject; and use the point-cloud representations of the subjectto generate respective virtual skeletons.
 18. The computer-readablemedia of claim 15, wherein a first of the three-dimensional images ofthe subject is captured using a time of flight device.
 19. Thecomputer-readable media of claim 15, wherein a first of thethree-dimensional images of the subject is captured using a mobilecommunication device camera.
 20. The computer-readable media of claim15, wherein a first of the three-dimensional images of the subject and asecond of the three-dimensional images of the subject are capturedwithin a week of each other.
 21. The computer-readable media of claim15, wherein a first of the three-dimensional images of the subjectcomprises an image of the subject and a background of known dimensions,the operations further comprising: acquire distance values associatedwith depth information for the first of the three dimensional images ofthe subject on the background; and acquire amplitude values associatedwith brightness information of reflected light for the first of thethree dimensional images of the subject on the background.